fix(azure): preserve membership consistency - #11309

Code coverage

Report-only conclusion: current-main baseline stale.

The newest successful coverage run tested 5267011, not current main 53b0307.

Coverage combines every CI test matrix job, including providers, CodeGen, .NET 8/10, Linux, Windows, and macOS, using canonical physical source and branch identities.

The comparison remains report-only while normal line and branch variance is calibrated.

Addressed the cleanup review in e4458f6. Cleanup now passes the selected rows to the existing DeleteEntriesBatch helper, reusing one fenced snapshot across all successful batches. Row-etag contention triggers a fresh selection after outstanding batches complete. Conditional deletions continue to protect concurrent heartbeats and votes.

The regression checks exactly one partition query and two version reads for 195 deletions with batch limits 1, 2, 3, and 100. The contention case now covers a successful batch followed by a conflicting batch, then confirms that refreshed liveness prevents deletion on retry.

I also consolidated repeated heartbeat-failure setup and empty/ineligible-cleanup cases, and reused the existing SDK mock factory. The PR is 57 net lines smaller while retaining all 76 focused cases. The net10.0 build and focused suite pass.

The small shared cleanup-test prerequisite remains necessary until #11308 lands; rebasing afterward will remove that fourth file from this PR's diff.

Follow-up to #11309 (review): the summary had no inline details, but the failure-handling audit identified and reproduced two concrete gaps. Fixed in 38cf2de:

• Concurrent cleanup now preserves every failed batch in an AggregateException and retries only when every failure is row contention. Mixed 412 + authorization/service failures propagate, regardless of batch order. Successful batches still reuse the same fenced snapshot.
• Conditional updates use GetEntityAsync, letting the real SDK distinguish missing rows (conditional failure) from TableNotFound (propagated infrastructure failure).

Regressions: CleanupPropagatesMixedBatchFailures, CleanupRetriesWhenEveryBatchFailureIsContention, and UpdateDistinguishesMissingRowFromMissingTableThroughSdk. The latter uses the actual TableClient with an in-memory HTTP transport, exercising SDK 404 classification. Existing tests continue to cover valid cleanup batch limits 1/2/3/100, recency/cutoff equality, cancellation, and row-etag safety; the review summary supplied no additional concrete validation requirement.

The new regressions reproduced five failures before the fix. The revised focused net10.0 class passes 84/84, with a clean build. Both Azure Storage CI jobs must pass on this exact head before this follow-up is considered complete. The PR remains draft.

The required Azure CI gate is now satisfied for exact head 38cf2de61751b0a60fa857a529f664ffcbe500f3:

• Azure Storage provider tests (net8.0): success.
• Azure Storage provider tests (net10.0): success.

This supplements the 84/84 focused regression run and clean local build. The PR remains draft; other checks are still running.

Simplified heartbeat writes in cca608bee22edac1deef20cfb83e066dfa01d658 using the owning-silo invariant: each heartbeat is a single unconditional merge containing only IAmAlive plus entity keys. The normal path performs zero reads. A missing retired row still triggers the canonical-history check before completing, and infrastructure failures propagate.

Removed the unnecessary heartbeat read/max/CAS retry loop and its stronger multi-writer assumptions from the tests. Full-row membership updates retain their separate protection against overwriting a newer heartbeat. The patch removes 66 net lines.

OwnerHeartbeatUsesSingleUnconditionalLivenessMerge asserts one SDK call, wildcard ETag, merge mode, and the liveness-only payload. Cancellation, missing-row/history, infrastructure errors, full-row safety, and cleanup regressions remain covered. Focused net10.0 result: 80 passed, zero failed/skipped; build clean. Both Azure Storage CI jobs must succeed on this new head before validation is complete.

Applied the clarified owner-only heartbeat policy in ebc225082c1a8f19331201e341f49699fa8a321a: UpdateIAmAlive directly returns the one wildcard IAmAlive-only merge. All native errors, including missing-row404, propagate. Removed the canonical-history probe and the post-retirement no-op assumption.

OwnerHeartbeatUsesSingleUnconditionalLivenessMerge, HeartbeatFailuresPropagateFromSingleMerge, and HeartbeatMergeHonorsCancellation assert the one-call path on success, native errors, and cancellation. Full-row membership max/etag protection and cleanup/read consistency fixes remain unchanged.

Focused net10.0 result: 78 passed, zero failed/skipped; build: zero warnings/errors. PR description updated. Validation now follows this exact head; awaiting both Azure Storage jobs and remaining checks.

Applied the clarified heartbeat-independent concurrency contract in 29b652a7f460195d7e39a8be3afe8bae7a4e5c35.

ReadRow/ReadAll now expose the canonical table-version etag as the opaque membership row token. UpdateRow validates logical row/table token agreement, then issues one atomic transaction using wildcard UpdateReplace for the existing silo row and CAS for the canonical version row. Removed the row preread, heartbeat maximum comparison, and physical heartbeat-etag dependency. IAmAlive follows the accepted best-effort semantics; heartbeat publication remains one blind column merge. Raw storage etags remain internal for conditional cleanup.

OwnerHeartbeatPreservesCanonicalTokensForMembershipUpdate exercises both read APIs: an owner heartbeat changes the physical etag, the original logical tokens remain valid, an IAM-only etag change during the read fence triggers no retry, and a status/vote update using the original tokens succeeds. The test also permits the full-row update to replace the heartbeat timestamp. Stale logical-token rejection, native transaction conflict/missing-row failure, and SDK-level table failure regressions remain covered.

Focused net10.0 validation: 78 passed, zero failed/skipped; clean build. PR description reflects the latest contract. Awaiting exact-head Azure Storage net8/net10 and remaining CI checks.

Audited review #11309 (review). Its sole linked finding asks for stored-heartbeat maximum preservation, which the final agreed contract explicitly supersedes. Answered and resolved that thread with the updated #11308 contract and the native accepted-overwrite regression: #11309 (comment).

The overview's additional references to cleanup filtering and token test expectations provide no separate location or reproduction. Current focused regressions cover Dead-only/recency/cutoff filtering, conditional cleanup races, canonical-token stability across heartbeats, and true stale-token/missing-row failures. There is no additional concrete change to make from those phrases. The latest 78-case focused suite passes; exact-head CI remains the completion gate. No code change was needed for this contract clarification.

Final-head validation completed for 29b652a7f460195d7e39a8be3afe8bae7a4e5c35:

• Azure Storage net8.0: success.
• Azure Storage net10.0: success.
• All currently reported checks: 70 success, 1 skipped, zero pending/failed.
• Focused native regression class: 78/78 passed; build clean.

The heartbeat-preservation review thread is answered and resolved against the final canonical-token contract. The branch/worktree is clean; no merge was performed.